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This paper describes a project designed to evaluate a 5-day clinical supervision workshop. The failure of 
Kirkpatrick’s 4-level evaluation model to include key contextual input variables is demonstrated to be a 
critical evaluative shortcoming. The project shows how the use of valid diagnostic tools like the Learning 
Transfer Systems Inventory can help evaluators overcome this fatal flaw, gain insight into key factors 
influencing training effectiveness, and make competent recommendations for improving training and 
learning transfer. 
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The Kirkpatrick four-level evaluation model (1994) has served as the primary organizing framework for training 
evaluations for over 40 years. There is no doubt the model has made valuable contributions to training evaluation 
thinking and practice. It has helped focus training evaluation practice on outcomes (Newstrom, 1995) and 
underscored the importance of examining multiple measures of training effectiveness. In particular, the distinction 
between learning (level two) and behavior (level three) has drawn increased attention to the importance of the 
learning transfer process in making training truly effective. 

However, we take the position that training is only effective if what is learned in training is ultimately 
applied on-the-job. If this fundamental criterion for training effectiveness is accepted then the 4-level model's 
exclusive focus on training outcomes becomes a critical evaluative shortcoming. By focusing only on learning, job 
behavior and so on the 4-level model provides an oversimplified view of training effectiveness because it doesn't 
consider a variety of critical input factors that can influence those outcomes. For example, a broad stream of 
research over past two decades (Salas & Cannon-Bowers, 2001; Tannenbaum & Yukl, 1992) has documented the 
presence of a wide range of organizational, individual, and training design and delivery factors that can influence 
training effectiveness before, during, or after training. This research has led to a new understanding of training 
effectiveness that considers work environment, organizational and individual factors as crucial input factors. 
Kirkpatrick’s model ignores these factors and, in effect, assumes their examination is not essential for effective 
training evaluation. But without understanding and measuring these factors as a part of evaluation it is not possible 
to fully understand why training is or is not effective. And, equally important, it is not possible to make competent 
recommendations about how training - and particularly the application of training on the job - can be improved. 

This paper describes a project designed to evaluate the learning and transfer of learning following 
participation in a 5-day clinical supervision workshop. For HRD practitioners, this project is valuable because it 
demonstrates how learning and learning transfer can be assessed using relatively straightforward pre- and post- 
training measures. More importantly, the project exemplifies why Kirkpatrick's 4-level model needs to be 
supplemented with valid, research-based, theoretically driven diagnostic tools that can provide information about 
factors that can influence training effectiveness. The Learning Transfer Systems Inventory (LTSI) was used as such 
a diagnostic in this project. It provided key insights into factors affecting training effectiveness and, in so doing, 
enabled evaluators to make accurate assessments of gaps in training system performance and to develop competent 
recommendations for improving training and learning transfer. 

Project Background 

The goal of the Addiction Counseling (AC) division in the Department of Health is to provide prevention and 
treatment services delivered by highly trained professionals. For this reason, the training and continuous skill 
upgrading of AC’s professional staff has become a high priority. Because of the substantial investments in training 
made by AC, there is a growing interest in taking steps to make training as effective as possible. This paper 
describes the evaluation of a five-day Clinical Supervision Training Workshop (CSTW). The workshop was 
conducted by an external provider and was aimed at improving the clinical supervision skills of a group of AC 
addiction counselors. The goal of the evaluation was to assess the training outcomes resulting from the CSTW and 
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to make recommendations about how training effectiveness might be improved. The evaluation focused on four 
questions: 

• To what extent did learning by clinical supervisors occur as a result of participation in the CSTW? 

• To what extent did clinical supervisors transfer knowledge and skills learned in the CSTW to job behavior? 

• What catalysts and barriers exist to on-the-job use of the knowledge and skills acquired in the CSTW? 

• What change is needed to help overcome the barriers to training transfer and to improve the on-the-job use 

of the knowledge and skills acquired in training? 

Methodology 

The Sample. Data for this evaluation were collected from 28 clinical supervisors. Those responding 
represented six different AC offices spread throughout a state in the southern US. Nearly all of clinical supervisors 
(96.4%, n = 27) had six or more years of experience with AC. About 64% (n = 18) had received less than eight 
hours of clinical supervision training prior to the present CSTW. Most respondents (67.9%, n = 19) spent less than 
50% of their time in staff supervision. Eleven (39%) of the participants were men. 

Data Collection 

Pre- and post-training learning and job behavior measures were collected to gauge learning and learning transfer. 
Critical factors affecting training effectiveness were also assessed using quantitative and qualitative methods. 
Obtaining this data required the development and use of multiple assessment instruments. 

The Job Behavior Rating Instrument The Job Behavior Rating Instrument (JBRI) was designed to assess 
respondents’ perceptions of the extent to which a subset of skills or behaviors taught in training were used by 
clinical supervisors as a part of their active clinical supervision practice. The JBRI therefore focused on a subset of 
five of the 20 modules delivered during CSTW. The five modules were selected with the assistance of several 
subject matter experts who reviewed the CSTW Manual and identified those modules most critical for effective 
clinical supervision. This subset of modules was then reviewed by the CSTW trainer to identify a) behavioral 
products that would be generated by a clinical supervisor using what he or she learned in training (e.g., develops 
practical treatment options); and b) process, principles, steps or procedures learned in training that should be 
exhibited on the job. The result was a list of 25 behavioral outcome statements that were subsequently reviewed and 
revised by the authors into a set 33 rating items. 

Assessment ratings were collected at two points in time. A pre-training JBRI was administered 10 days 
before training and a post-training JBRI was administered three weeks after training. Ratings were collected from 
two sources: self-ratings by clinical supervisors about their use of the learned skills or behaviors, and subordinates’ 
ratings of their supervisors’ skill/behavior use. Respondents were asked to provide a rating of the extent to which 
the clinical supervisor used each skill or behavior as a part of his or her current clinical supervision practice. 
Ratings were provided along a 5-point Likert-type scale: 1 = never, 2 = almost never, 3 = sometimes, 4 = very often, 
5 = always. For each administration, the JBRI was emailed to respondents with a cover letter explaining the project 
and the purpose of the instrument. Several follow-up emails were sent to encourage respondents to complete the 
instrument. Completed instruments were returned directly to the lead investigator via email or post. 

The Learning Assessment Instrument The Learning Assessment Instrument (LAI) was a 39-point test 
designed to provide a reliable measure of learning gains from training. Items on the LAI were drawn from 14 of the 
20 modules included in the Clinical Supervision Workshop with 20 items representing the five critical modules 
targeted by the JBRI. The remaining 19 items represented nine other modules. A pre-training administration of the 
LAI was completed at the beginning of the first day of the CSTW and a post-training administration was completed 
late on the final day of training. 

The Learning Transfer Systems Inventory The Learning Transfer Systems Inventory (LTSI) was used to 
provide information about characteristics of the workplace, training design and content, individual attributes, and 
groups variables that can act as either barriers or catalysts to training transfer. The LTSI is based on Noe’s theory of 
work behavior (1986) and is built around a comprehensive evaluation and measurement model (Holton, 1996). 
Conceptually, the LTSI views performance improvement from training as a function of a system of factors 
represented by four sets of elements: secondary elements, ability/enabling elements, motivation elements, and 
environmental elements (see Figure 1). Nested within these four domains are measures addressing a set of 16 
learning transfer system constructs. 
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Figure 1 : Conceptual Framework for the LTSI 
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The LTSI is a survey instrument and is currently the only validated diagnostic tool available that measures 
a comprehensive set of learning transfer system factors. It has undergone several developmental iterations and 
research has provided evidence of the instrument’s construct (Holton, Bates, & Ruona, 2000; Bookter, 1999) and 
criterion-related validity (Bates, 2001). Recent research also suggests the LTSI may have cross-cultural 
applicability (e.g.. Bates, Kauffeld, & Holton, 2004; Chen, Holton, & Bates, in press; Khasawneh, Bates, & Holton, 
in press). 

For this project, the LTSI contained 74 items and was divided into three sections. The first section 
contained 45 items measuring 11 transfer constructs specifically referencing the CSTW. For this section, 
respondents were directed “to think about the Initial Clinical Supervision Workshop" when responding to these 
items. The second section contained 23 items measuring five constructs that reflect respondents’ general experience 
with training in ADC. Respondents were directed to “think about training in general in your organization” when 
responding to these items. For items in section 1 and 2 respondents were asked to rate the items on a Likert-type 
scale ranging from 1 = strongly disagree, 2= disagree, 3 = neither agree nor disagree, 4 = agree, to 5 = strongly 
agree. The final section of the survey contained 6 items directed at collecting a variety of descriptive information 
about the sample. 

Focus Groups A series of three focus groups were conducted with different groups of clinical supervisors 
four to five weeks following training to obtain feedback about training content and processes, collect in-depth 
information about learning transfer system catalysts and barriers, and gather insights for training and learning 
transfer improvement. A total of 13 clinical supervisors from three different regional offices participated in the 
group sessions. 

Data Analysis 

The LAI Responses to the LAI were evaluated and the number of correct responses made by each 
respondent were calculated. Correct response scores on the pre- and post-training administrations were compared 
to evaluate whether and to what extent the scores changed. A paired-samples t-test procedure computed the 
differences between the two test scores for each training participant and tested whether the overall average differed 
significantly from 0. 
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The JBRI Mean self- and subordinate pie- and post-training job behavior ratings across 33 JBRI items 
were calculated. The pre- and post-training means for each of the 33 items were compared separately for clinical 
supervisor self-ratings and for subordinate ratings and a difference score for each was calculated. The difference 
scores were used to demonstrate the extent to which pre- and post-training ratings changed for each group of raters. 
A paired-samples t-test procedure was run for each group of raters that computed the differences between the two 
ratings for each JBRI item and tested whether the overall average differed significantly from 0. 

The LTSI. Scale reliabilities were estimated using Cronbach’s alpha, a measure of the internal consistency 
of items in a scale. Scale scores were then computed for each scale for each respondent by calculating the mean 
rating for all items in a scale. The scale scores were then averaged across all respondents giving a single grand 
mean score for each scale. 

Focus Group Data Focus group sessions were tape recorded and transcribed. Transcriptions and 
facilitator notes from the sessions were analyzed using a constant comparative method to identify topics, 
interconnections, and themes about the predominant aspects of training effectiveness and the learning transfer 
system catalysts and barriers facing clinical supervisors in AC. Four key themes emerged from this data. First, data 
indicated that, in general, trainees felt more should be done to prepare them to enter, participate, and learn in 
training. For example, respondents wanted more information about why they had been selected for training and how 
the content of training would benefit their job-related development. Second, data indicated that respondents would 
have benefited from more direct supervisor involvement and support in preparing for training and in using the new 
learning once back on the job. Third, respondents indicated that much of the training they had attended in the past 
often did not directly address job requirements or performance problems they felt they were faced with. Finally, 
respondents indicated there was typically little or no available “space” in their daily work schedules to allow for 
opportunities to use new learning and, when opportunities did arise, that efforts to use new learning should be more 
systematically rewarded. 

Results 



Question 1: To what extent did learning by the clinical supervisors occur as a result of participation in 
the CSTW? The data indicated that 27 of 29 supervisors improved their scores on the post-test over those achieved 
on the pre-test. The post-test score for one participant decreased and the score for one participant did not change 
from the pre-test score. Table 1 shows the overall pre- and post-test means, standard deviations, and the results of a 
paired sample test of difference in pre- and post-test scores. Mean test scores across all participants increased from 
17.76 on the pre-test to 24.21 on the post-test, a statistically significant gain of 6.45 points. In short, there appears to 
have been a notable increase in learning from the pre-test to the post-test for nearly all of the training participants. 



Table 1: Test of Pre- and Post-Training Learning Assessment Means 





Mean 


N 


Sid. Deviation 


Paired Sample T-Test 


Pre-Test 


17.76 


29 


4.46 




Post-Test 


24.21 


29 


3.78 




Difference 


6.45 


29 


4.30 


8.08* 



*p < .05 



Question 2: To what extent did clinical supervisors transfer the skills learned in the CSTW to job 
behavior? The data for this question was based on the average self-ratings of 28 clinical supervisors and the 
average ratings of supervisor behavior from 44 subordinates across 33 behavioral elements. Table 2 shows the 
overall self- and subordinate pre- and post-training JBRI means and the results of a paired sample test of difference 
between pre- and post-training ratings. Results show that the average self-rating of clinical supervisors showed a 
slight increase (+.12) in target skills and behaviors whereas average subordinate ratings showed a slight decrease 
(-.08). Both differences, although modest, were significantly different from 0. 

Closer inspection of the data generated from the JBRI reinforces this mixed view of the extent to which the 
skills and behaviors learned in training were transferred by clinical supervisors. On one hand, self- and subordinate 
ratings agreed in showing an increase in the use of 7 of the 33 (21.2%) behavior elements. On the other hand, 
ratings from both sources showed a decrease in the use of 8 of the 33 (24.2%) behavior elements. For the remaining 
18 (55.6%) behavior elements the average self-ratings of the clinical supervisors indicated some increase in use 
whereas subordinate ratings indicated a decrease in use. In sum, using the agreement of self- and subordinate ratings 
as the strongest evidence of learning transfer, the data suggested that there was some job application of a modest 
proportion (21%) of the skills and behaviors assessed for this project. For the remaining 79% of the behavior 
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elements there was no clear evidence that these behavior elements showed any substantial increase in use on the job 
following training. 



Table 2: Test of Self- & Subordinate Pre- & Post-Training Job Behavior Rating Means 





N 


Mean 


Mean Difference 
(Post - Pre) 


Paired Sample 7- 
Test 


Self-Ratings 

Pre 


28 


3.51 






Post 


28 


3.63 


.12 


3.61* 


Subordinate Ratings 










Pre 


44 


3.20 






Post 


44 


3.12 


-.08 


3.43* 



*p < .05 



It is important to note that had this evaluation stopped at this point with the assessment of learning and job behavior 
outcomes we would have known only that very modest learning occurred and that the transfer of learning to job 
behavior three weeks after training was weak. We would not, however, have known anything about why training 
was not more effective or about how it might be made more effective in the future. This is a crippling flaw of the 4- 
level approach to evaluation. It is also the point at which taxonomic evaluation models such as Kirkpatrick's need 
to be supplemented with additional tools and models so that critical influences on training effectiveness can be 
assessed and analyzed. The following steps demonstrate how this can be done using valid quantitative diagnostic 
tools (the LTSI) supplemented with qualitative data gathered through focus groups. 

Question 3: What catalysts and barriers exist to on-the-job me of the knowledge and skills acquired in 
the CSTW? Responses to the LTSI coupled with qualitative data from the focus groups were used to answer this 
question. Scale means from the LTSI were rank ordered from highest to lowest and categorized according to the 
following criteria: 

< 2.5 Severe Barrier to Learning Transfer 

2.51 - 3.5 Barrier to Learning Transfer 

3.5 1 - 4.00 Weak Catalyst for Transfer Needing Improvement 

> 4.01 Catalyst for Learning Transfer 

The prioritization and categorization process was done as a practical aid in identifying leverage points for change in 
the learning transfer system facing clinical supervisors. Results (Table 3) indicated that barriers to transfer far 
outnumbered the catalysts. Eleven of 15 scales emerged as either barriers or severe barriers to learning transfer, 3 
scales as catalysts needing improvement, and one scale as a catalyst for transfer. One scale was deleted from the 
analysis because of low reliabilities for this sample. When interpreted in conjunction with the focus group data 
these findings have the following implications for training effectiveness. 

The Catalyst: Transfer Design The relatively high score for Transfer Design (4.10) reinforces focus group 
data indicating that trainees viewed the manner in which the training content was delivered facilitated learning 
transfer. Training participants felt that the use of role plays, cases, scenarios and examples, and directed group work 
helped to both clearly link learning with on-the-job performance and to demonstrate and provide practice in how to 
apply what was being learned. 

Potentially Powerful Catalysts: Trainee Motivation The Clinical Supervisors’ confidence in their own 
ability to improve job performance (Performance Self-Efficacy), their level of motivation to transfer learning, and 
their expectations that devoting energy to applying training will result in performance improvements (Transfer 
Effort - Performance Expectations) emerged as positive transfer variables. Scores on these motivational variables 
suggested that the clinical supervisors were relatively well motivated to use training to improve their job 
performance. These motivational factors represented a potentially powerful catalyst for training effectiveness if 
some of the barriers could be removed or reduced. 

Substantial Barriers to Transfer: Consequences, Interpersonal Support, & Opportunity to Use The most 
substantial barriers to learning transfer ultimate training effectiveness appeared to reside in the work environment of 
the clinical supervisors. The two factors with the lowest scale scores. Personal Outcomes Positive and Personal 
Outcomes Negative, relate to the rewards, recognition or other consequences clinical supervisors perceived as tied to 
the presence or absence of learning transfer. The data suggested that there are neither positive nor negative 
consequences that attended the learning transfer efforts of clinical supervisors. Data also indicated that the level of 
active interpersonal support for transfer, particularly from trainees’ supervisors or managers, was limited at best. 
Although supervisors/managers provided substantial verbal encouragement promoting training attendance, the level 
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Table 3: LTSI Scale Scores Rank Ordered and Classified 

LTSI Scale Name Mean LTSI Scale Description 

Catalysts for Learning Transfer (> 4.01) 

Transfer Design 4.23 The extent to which training has been designed to give trainees the ability to transfer learning to job 

application. 

Catalysts Needing Improvement (3.51 - 4.00) 

Motivation to Transfer 3.91 The direction, intensity and persistence of effort exerted toward using on the job skills and knowledge learned 

in training. 

Transfer Effort- 3.87 The expectation that effort devoted to transferring learning will lead to changes in job performance. 

Performance Expectations 

Performance Self- 3.58 An individual’s general belief that he/she is able to change his/her performance when needed or desired. 

Efficacy 

Barriers to Learning Transfer (2.5 - 3.5) 

Content Validity 3.45 The extent to which the trainees judge the training content to accurately reflect job requirements. 

Learner Readiness 3.29 The extent to which individuals are prepared to enter and participate in a training program. 

Peer Support 3.24 The extent to which a trainee’s peers reinforce and support use of learning on-the-job. 

Opportunity to Use 3.23 The extent to which trainees are provided with or obtain resources and tasks on the job that enable them to use 

the skills taught in training. 

Openness to Change 3.16 The extent to which prevailing group norms resist or discourage the use of new skills and knowledge acquired 

in training. 

Performance-Outcome 3.04 The expectation that changes in job performance will lead to outcomes valued by the individual. 

Expectations 

Performance Feedback 2.93 Formal and informal indicators from an organization received by an individual about his or her job 

performance. 

Personal Capacity 2.87 The extent to which individuals have the time, energy and mental space in their work lives to make changes 

required to transfer learning to the job. 

Supervisor Support 2.75 The extent to which the trainee’s supervisors/managers support & reinforce the use of learning on-the-job. 

Severe Barriers to Learning Transfer (< 2.5) 

Personal Outcomes 2.20 The extent to which individuals believe that if they do not apply new skills and knowledge learned in training 

Negative that it will lead to outcomes that are negative or undesirable. 

Personal Outcomes 1.93 The degree to which applying training on the job leads to outcomes that are positive or desirable for the 

Positive individual. 

of active assistance in helping the clinical supervisors apply training on the job needed substantial improvement. In 
addition, data indicated that the clinical supervisors’ supervisors or managers often lacked knowledge about the 
content or utility of training. Consequently there was virtually no communication from trainees’ superiors about the 
value of training or learning transfer expectations or goals, no evaluation of learning transfer or feedback discussions 
about transfer successes or obstacles to success. The need for enhanced levels of interpersonal support is also 
evident in the scale scores for Peer Support and clinical supervisors desire to network with colleagues, after training 
and following a return to work, for advice and support in the application of new learning. Relatively low scores on 
the Openness to Change and Performance Feedback reinforce these findings. 

Qualitative and quantitative data also indicated that the clinical supervisors typically had little time or 
opportunity to experiment or try to use new learning when they returned to work after training. This finding was 
evident in low scores for Personal Capacity for Transfer and Opportunity to Use, in addition to comments collected 
in focus groups. Work pressures, time constraints, backlog of work resulting from training attendance, and 
administrative duties were some key constraints that limited clinical supervisors’ capacity to try to use new learning 
back on the job. 

Evidence for another critical barrier to transfer emerged from a consolidation of focus group themes and LTSI 
data. Discussions with clinical supervisors suggested the need for a more systematic and effective front-end analysis 
process in AC. Front-end analysis of training needs or transfer obstacles seldom occurred or, when it did occur, it 
was limited to irregular “felt-needs” surveys. These factors coupled with the number of barriers to learning transfer 
identified in this assessment strongly suggested the need for a systematic front-end analysis process in AC. Such a 
process is fundamental to effective training and transfer because it provides the information for building 
performance-based training and an appropriate configuration of systems, policies, processes, and other work-related 
factors to support the application of new learning. 

Training Process Barriers: Content Validity and Learner Readiness These two factors relate to the 
training process and emerged as somewhat less intense barriers to learning transfer than those that have been 
addressed so far. The mean score on the Content Validity scale (3.45) suggested the factor may represent a barrier 
to transfer. Although focus group participants were fairly positive in their comments supporting the job relevance of 
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this training, their comments also indicated that the training might have suffered under the weight of the large 
amount of information presented. Focus group comments indicated that, because of the sheer volume of information 
presented during the 5-day training, trainees had to selectively filter through it and focus on those pieces they saw as 
most relevant and useful to their immediate individual needs and challenges. This selective filtering process, by 
emphasizing the value of a limited segment of the content, apparently led to a decrease in the overall perceived job 
relevance of the training. A modest mean score on the Learner Readiness scale (3.29) suggested that clinical 
supervisors could have been better prepared prior to the beginning of the Workshop to understand and gain value 
from this training. One of the focus group themes to emerge indicated that the clinical supervisors did not receive 
sufficiently clear information about the specific knowledge and skills they should expect to learn or how that 
learning would improve performance. 

Summary 

The quantitative and qualitative data collected during the course of this project depicted a situation in which a group 
of relatively well motivated clinical supervisors participated in a 5-day training workshop that they felt was 
delivered in ways that facilitated learning transfer and was, at least in part, seen to be relevant to their jobs. Data 
indicated that modest learning gains resulted from training participation. And, after three weeks back on the job, 
only a relatively small percentage (21%) of the most critical knowledge and skills included in the training were 
being used. Finally, it is clear from the data that clinical supervisors faced a number of barriers to more effective 
training and comprehensive learning transfer. 

Based on these findings the following recommendations were directed at key leverage points for improving 
overall training effectiveness: 

1. Develop and implement a systematic front-end analysis process to identify performance -related training needs 
and diagnose transfer problems before training is designed and delivered. 

2. Implement a process for communicating to all key stakeholders in the training effort including managers, direct 
supervisors, and trainees the content, rationale for, and desired outcomes of training before training begins. 

3. Build the capacity of managers and direct supervisors to actively support and facilitate learning transfer. 
Priority areas include: 

’ Prepare managers and direct supervisors with a sound and fundamental understanding of learning transfer 
and the factors that can influence it in the workplace. 

■ Implementing an informal reward system that supports the job application of learning. 

’ Developing the capacity of managers and direct supervisors to provide ongoing coaching and feedback for 
learning transfer, including the use of goal-setting and action planning for learning and transfer. 

4. Implement post-training mechanisms so trainees can network and collaborate with peers and colleagues about 
learning and training transfer. 

5. For training that is important, develop and implement a process for evaluating training and assessing learning 
transfer. At a minimum, the evaluation process should provide information about the: 

■ Job-relevance of training, the quality of training delivery, and the transferability of learning. 

" Learning that occurred as a result of training participation. 

" Performance/job behavior changes that resulted from the application of learning. 

’ Factors influencing the job application of training. 

Implications for Practitioners 

In general, the benefits that can be derived from training evaluation are directly related to the evaluation models, 
methods, and tools used and their capacity to provide information that increases the clarity of judgment and reduces 
the uncertainty of action for our clients and stakeholders (Patton, 1997). For training evaluation, this is best done 
when quantitative and qualitative, descriptive and judgmental information is systematically collected in ways that a) 
assess program effectiveness, and b) helps improve the program relative to its goals (Goldstein & Ford, 2002; 
Holton, 1996). Unfortunately, the failure of Kirkpatrick’s 4-level model to include consideration of key contextual 
input variables in training evaluation masks the real complexities of the training process. The problem with using 
the 4-level model is that, although it may provide some beneficial information about program outcomes (assuming 
appropriate criteria are selected and adequately measured), when measurement is restricted to training outcomes no 
information about why training was or was not effective is generated. In the present case study, it would have been 
impossible to understand the barriers to training effectiveness (including transfer) and to subsequently make relevant 
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and effective recommendations by measuring outcomes alone. Examination of a relatively broad range the learning 
transfer system variables provided a far more complete and accurate picture of the gaps in training effectiveness. In 
other words, the danger is that practitioners using the 4-level approach alone will quite likely remain woefully 
uninformed about critical aspects of training effectiveness, will consequently arrive at erroneous conclusions about 
their training programs (Holton, 1996). This case demonstrates the value of diagnostic tools like the LTSI in 
training evaluation as critical supplements to traditional outcomes-focused evaluations. Tools such as this provide a 
theoretical framework for understanding training effectiveness and furnish valid and reliable means for assessing 
factors essential to training effectiveness. For evaluators, the result is a more complete and accurate understanding 
of the ‘why and wherefore’ of evaluative outcomes, and a firm foundation for making useful recommendations for 
improvement. 

This case has demonstrated the value of the LTSI as an evaluative tool capable of identifying leverage 
points for change in the training - learning transfer - performance improvement process. These leverage points are 
factors that are either facilitating or blocking training effectiveness. There are at least two approaches to the 
interpretation of these leverage points (Holton, 2003) that should be examined in future research and practice. This 
first, the fit approach, suggests that the critical leverage points are likely to be a function of the absolute level of a 
particular factor and its salience in a particular organization’s culture. For example, in the present case supervisor 
support issues were quite important. In a more team-oriented culture, peer and co-worker related support variables 
may have been more important. The second approach, the multiple configurations approach, suggests that it is not 
any one or two particular factors that are critical. Rather, what is important is the total overall quality of the learning 
transfer system. From this perspective, the transfer factors are seen as operating as a system to support training 
effectiveness. In this view, weak elements may be offset or compensated by stronger elements. For example, in the 
low levels of supervisor support may be offset by more supportive reward systems and stronger transfer designs. In 
other words, instead of only a single answer to more effective training there are multiple options or configurations of 
learning transfer factors that can maximize training effectiveness. Future research and practice with the LTSI as an 
evaluative tool should examine the utility of these separate approaches. 
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